Shore Side Data System (600031 and 900823) : Internal Application Consolidation

This page last changed on Jun 09, 2008 by kgomes.

In order to get our local (MBARI) installation of SSDS in a manageable state, I went through an application consolidation phase to try and clean up a bunch of stuff. The first thing to do was to create a layout of how things are now.

And then a diagram of how they will look after the cleanup:

And then the steps on how to make that transition:

ssdspub.mbari.org

The easiest place to clean first, was the machine ssdspub.mbari.org. Currently it is basically just serving the purpose of a tomcat container. There are still services out there, but they are not really serving any purpose since they are pointed to a database that is defunct. To clean up, I did the following:

I first removed the axis.war file from the deploy directory.
I then removed the omse.war and the mse.war web applications.
Move MSE to the inside?

I am wondering if I shouldn't move the mse.war pages to the new-ssds.mbari.org server so they are at least available.

I then shutdown Jboss, removed access.war, ssds-data-mssql-ds.xml, ssds-mssql-ds.xml and ssds-services-ssdspub.jar

access.war wasn't so simple

When I removed access.war, it messed up some people who were using the old GetOriginalDataServlet and the forwards from the old /access/*.jsp's were broken. I put an access.war back out there, but removed the servlets and put notes on the other pages that said either the pages were no longer available or where they could go to get to them.

I then deployed access.war and cimt.war on to new-ssds.mbari.org (to prepare for the CNAME change)
I then restarted JBoss
I also updated the index.html page in the apache installation to point to the cimt web application so that if people go to ssdspub.mbari.org they will see something.
I had Neil shut off the replication jobs that were rebuild the SSDS database on ssdspub each day.
I also had Todd and Neil shut off the replication jobs that were copying the raw data files from bob.shore.mbari.org, iagdata share on tornado, and the ssdsdata share on tornado out to SSDSPub as they are no longer needed.
I then set the MSSQLServer and SQLServerAgent service to 'Manual' and shut them off.
Get rid of SSDSPUB?

In theory, I should now be able to remove ssdspub.mbari.org if I CNAME it to new-ssds.mbari.org

predator.shore.mbari.org

Next, I could do a similar cleanup of predator.
First, I removed axis.war
Then I removed mtm3.war
Now, my current thinking is that instead of going through the database and changing everything under the sun, can I just change the CNAME of ssds.shore.mbari.org to point to new-ssds.mbari.org. In order to do that, I need to:
1. Change all references from predator.shore.mbari.org to ssds.shore.mbari.org in DataContainer.uriString, Resource.uriString and Software.uriString and make sure those entities exist.
  1. First I queried to find all the DataContainers with predator in their URIString. I got back 23 rows of DataContainers whose uriStrings are no longer valid. Since this is the case, there will be no harm in just changing them with the following SQL:
    UPDATE ssdsdba.DataContainer SET uriString = REPLACE(uriString, 'predator.shore', 'ssds.shore') WHERE uriString like '%predator.shore%'
  2. Next thing was to do it for the Resources. Now, here there was a small snag. Some of the old NetCDF logs have an analogous entry for ssds.shore already so when the update was tried, I got duplicate unique key constraint violations. So, first I just searched for entries that pointed to the ssds/xml directory.
    SELECT * from ssdsdba.Resource where uriString like '%predator.shore.mbari.org/ssds/xml%'
    This returned 47 rows and they seemed to be valid uriStrings even though they were from really old stuff. So, I simply changed the uriString to point to ssds.shore instead of predator with the following:
    UPDATE ssdsdba.Resource set uriString = REPLACE(uriString,'predator.shore','ssds.shore') where uriString like '%predator.shore.mbari.org/ssds/xml%'
  3. After that, I queried for the other resources with predator in the name using:
    SELECT * from ssdsdba.Resource where uriString like '%predator.shore%'
    and it returned 24 rows of things that do not exist. Since they don't exist at the uri's and renamed hit unique key constraints, I just decided to remove them by first removing references to them in the assocResource tables.
    select * from ssdsdba.DataContainerAssocResource where ResourceID_FK in (select id from ssdsdba.Resource where uriString like '%predator.shore%') select * from ssdsdba.DataProducerAssocResource where ResourceID_FK in (select id from ssdsdba.Resource where uriString like '%predator.shore%') select * from ssdsdba.DeviceAssocResource where ResourceID_FK in (select id from ssdsdba.Resource where uriString like '%predator.shore%') select * from ssdsdba.SoftwareAssocResource where ResourceID_FK in (select id from ssdsdba.Resource where uriString like '%predator.shore%')
    The only one that found anything was for DataProducers (48 rows), so I removed all assoc records using:
    delete from ssdsdba.DataProducerAssocResource WHERE ResourceID_FK in (select id from ssdsdba.Resource where uriString like '%predator.shore%')
    Now that all the links to the resources with uriStrings with predator are removed, remove the resources themselves with:
    delete from ssdsdba.Resource WHERE uriString like '%predator.shore%'
    That removed 24 rows
  4. There were no uriStrings in the Software table that have references to predator.shore, so I did not do anything
2. Now that the predator name has been removed from the uriStrings, let's make sure there are no dods references in the uriStrings. I can search for those using:
```
SELECT * from ssdsdba.DataContainer where uriString like '%nph-dods%'
```
  That returned a whopping 1590 records, but there are basically two roots of the URLs that are of importance, they are:
```
http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/data/
```
  and
```
http://ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd/
```
  Since the auvctd ones are mapped through to the auvctd share on Tornado and the dods.mbari.org auvctd is the same, we can simply map the ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd to the dods.mbari.org machine using
```
UPDATE ssdsdba.DataContainer set uriString = REPLACE(uriString,'http://ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd','http://dods.mbari.org/cgi-bin/nph-nc/data/auvctd') where uriString like 'http://ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd%'
```
  Since the rest of the DataContainers that have uriStrings with nph-dods in them are pointing to old data and I can't rename them (they would create duplicate uriStrings because we used to put parallel dods and http file uris in there), I am just going to let them be and have broken links (for now). So there are 1255 records like that with broken links.
3. Verify all DODS urls are accessible through dods.mbari.org
  1. Currently, here is the list of DODS URLs that are available through ssds.shore.mbari.org:
    1. http://ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd/ (which is the mount of AUVCTD on Tornado)
    2. http://ssds.shore.mbari.org/cgi-bin/nph-dods/clients/ (which is a broken link)
    3. http://ssds.shore.mbari.org/cgi-bin/nph-dods/data/ (which is the mount to the data volume on bob.shore.mbari.org).
    4. http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/data/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/data/)
    5. http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/rawpackets/ (which is a link through the 'data' mount to the rawpacket on bob.shore.mbari.org)
    6. http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/rss/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/rss/)
    7. http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/transmogrify/ (which is a link through the 'data' mount to the transmogrify directory on bob.shore.mbari.org)
    8. http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/xml/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/xml/)
  2. Let's look at these on a case-by-case basis
    1. The AUVCTD mount on ssds.shore is the same as the one on dods.mbari.org. So the following URLs should be equivalent:
      http://ssds.shore.mbari.org/cgi-bin/nph-dods/auvctd/
      equals:
      http://dods.mbari.org/cgi-bin/nph-nc/data/auvctd/
    2. For the clients URL, since it is broken, there is no equivalent
    3. For the /data which is a mount to bob.shore.mbari.org, there is no equivalent URL on dods.mbari.org. That might be fine, we will find out in a minute.
    4. The /ssds/data URL on ssds.shore points to the ssds share on iagdata which is accessible through dods.mbari.org from the /data/ssds share. So these are equivalent:
      http://ssds.shore.mbari.org/cgi-bin/nph-dods/ssds/data
      equals:
      http://dods.mbari.org/cgi-bin/nph-nc/data/ssds/
      There is a problem though that on the dods.mbari.org side, there is a permissions denied in trying to access it. However, I don't think we really need this share and it would be nice to remove it if possible.
    5. That last one also applies to the rss and xml directories
    6. The ssds/rawpackets and transmogrify urls point to the raw packet and transmogrify share on bob and is not available through dods.mbari, but that should be OK. I will find out shortly.
  3. Now that we have an idea of how they are mapped, let's take a look at the DataContainer's and their base uriStrings to see if they point to any nph-dods urls. Since these are the same broken linked files that I found above and they cannot be mapped due to duplicate uriString constraint, I will just leave the uriStrings for DataContainers alone.
  4. For the DataContainer dodsUrlString, I can query to find any current dods urls that point to ssds.shore using:
    select * from ssdsdba.DataContainer where dodsUrlString LIKE '%nph-dods%'
    Since this returned no results, we should be fine on the data container side of things (I think we did that move earlier).
  5. We need to do the same for any resources we find and search the uriString for nph-dods:
    select * from ssdsdba.Resource where uriString LIKE '%nph-dods%'
    Which returned no results so we are good there.
  6. Also check software
    select * from ssdsdba.Software where uriString LIKE '%nph-dods%'
    Which also returned no results.
4. Now, we have all nph-dods urls that point to ssds.shore removed (except for the broken 1255) and a CNAME change should work if we point ssds.shore to new-ssds. Before we do that though, we must make sure all HTTP accessible shares on predator are available on new-ssds at the same base URL (i.e. new-ssds.mbari.org/ should be the equivalent of ssds.shore.mbari.org from an HTTP directory sharing standpoint. So, the following HTTP shares are available on ssds.shore:
  1. http://ssds.shore.mbari.org/auvctd/ (which is the mount of AUVCTD on Tornado)
  2. http://ssds.shore.mbari.org/clients/ (which is a broken link)
  3. http://ssds.shore.mbari.org/data/ (which is the mount to the data volume on bob.shore.mbari.org).
  4. http://ssds.shore.mbari.org/ssds/data/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/data/)
  5. http://ssds.shore.mbari.org/ssds/rawpackets/ (which is a link through the 'data' mount to the rawpacket on bob.shore.mbari.org)
  6. http://ssds.shore.mbari.org/ssds/rss/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/rss/)
  7. http://ssds.shore.mbari.org/ssds/transmogrify/ (which is a link through the 'data' mount to the transmogrify directory on bob.shore.mbari.org)
  8. http://ssds.shore.mbari.org/ssds/xml/ (which is a link through a mount to tornado.shore.mbari.org/iagdata/ssds/xml/)
5. So if we look at them one-by-one:
  1. http://ssds.shore.mbari.org/auvctd/ does not have an equivalent on new-ssds, but I have a trouble ticket into I.S. to get that mounted.
  2. http://ssds.shore.mbari.org/clients/ since it is a broken link, I am not worried about making it available through new-ssds.
  3. http://ssds.shore.mbari.org/data/ I am hoping to not have any links pointing to this, so hopefully I can not make that share available.
  4. http://ssds.shore.mbari.org/ssds/data/ I am hoping I can get rid of this
  5. http://ssds.shore.mbari.org/ssds/rawpackets/ I am hoping I can get rid of this
  6. http://ssds.shore.mbari.org/ssds/rss/ I am hoping I can get rid of this
  7. http://ssds.shore.mbari.org/ssds/transmogrify/ I am hoping I can get rid of this
  8. http://ssds.shore.mbari.org/ssds/xml/ I am hoping I can get rid of this
6. So let's start with the DataContainer uriStrings (I am going to ignore DODS URLs since they were done). I ran the following search:
```
select * from ssdsdba.DataContainer where uriString LIKE 'http://ssds.shore.mbari.org/auvctd%'
```
  I get 4692 results. As long as I can get the auvctd mount working on new-ssds, the CNAME should fix these.
```
select * from ssdsdba.DataContainer where uriString LIKE 'http://ssds.shore.mbari.org/clients%'
```
  This returned 0 results, so we are good to get rid of it.
```
select * from ssdsdba.DataContainer where uriString LIKE 'http://ssds.shore.mbari.org/data%'
```
  Again, 0 results.
```
select * from ssdsdba.DataContainer where uriString LIKE 'http://ssds.shore.mbari.org/ssds/data/%'
```
  Returned 1278 entries. All the other /ssds/* urls returned nothing so we are good there. Looking at the Resource table, it looks like there are uriStrings that point to /ssds/data and /ssds/xml, but they all look very out of date. There were no uriStrings in the Software table that pointed to the ssds.shore url so we are good there. So the big question becomes can we just remove all references to those old shares from the metadata since I think most of those have been reprocessed anyway? I have contacted Mike McCann about it. If that is the case I can get rid of:
  1. ssds share/url on dods.mbari.org
  2. All the dods and http share/urls from ssds.shore.mbari.org
  3. All of the data housed in the iagdata/ssds share on tornado
    Great, got the OK from Mike, so I can do all this and then we can go back and clean out the DB of any DataContainer, DataProducers, and Resources that are associated with these URLs. COOL!
  4. One small change, there are a handful of Resources that are XML files for data streams. Those might be useful, so I could copy those over the current Ruminate xml share on new-ssds and update the URLs to point to them there. Actually it looks like they have already been copied, probably when I moved to new-ssds, so I just need to update the URLs with:
    UPDATE ssdsdba.Resource set uriString = REPLACE(uriString,'http://ssds.shore.mbari.org/ssds/xml','http://new-ssds.mbari.org/data/ssds/ruminate/xml') where uriString like 'http://ssds.shore.mbari.org/ssds/xml%'
OK, now with that all cleaned up, I needed to move the existing updatebot, graphing and data checking perl script to the machine
```
pismo.shore.mbari.org
```
1. Pat was able to construct the same directory structures and permissions on pismo as on predator, so I just copied the files over to /opt/ssds and changed the scripts to point to the correct java locations.
2. With the data checking perl script, I changed it to point to the new-ssds GetOriginalDataServlet so that it would be reading the packets from the database and is a much better test as we can see the packets at the database and not just the file system.
Now with all things moved to pismo, we should be able to change the CNAME of ssds.shore.mbari.org to point to new-ssds.mbari.org instead of to predator.shore.mbari.org. Pete made that change and I shutdown the jboss (ssds) service and the httpd service on predator. I also commented out the various mounts in /etc/fstab so that the shares would no longer be mounted on predator.

new-ssds.mbari.org

Now, I currently have ruminate running on new-ssds as a message driven bean that is writing the XML files to a local directory /data/ssds/ruminate/xml. This really should be stored on the /tornado.shore.mbari.org/ssdsdata/ssds/ share under something like: /tornado.shore.mbari.org/ssdsdata/ssds/ruminate/xml. This means that I need to get a read-write share mounted from /tornado.shore.mbari.org/ssdsdata/ssds/ruminate that I can mount on new-ssds. If I can do this, I can then point any urls to the /ssdsdata/ssds/ruminate url on new-ssds and turn off the http share to the local /data/ssds/ruminate directory. The same goes for the /data/ssds/generated/gps directory.

There were security concerns (rightly so) about setting up a write share through the firewall, so instead, we setup a copy to run every 10 minutes and copy all files from the /data/ssds/ruminate/xml to the tornado /ssdsdata/ssds/ruminate/xml directories. Also, a similar copy was setup for /data/ssds/generated/gps. This means that any URLs that used to point to:

http://new-ssds.mbari.org/data/ssds/ruminate/xml

should point to

http://new-ssds.mbari.org/ssdsdata/ssds/ruminate/xml

and

http://new-ssds.mbari.org/data/ssds/generated/gps

should point to

http://new-ssds.mbari.org/ssdsdata/ssds/generated/gps

The SQL for that to happen is:

UPDATE ssdsdba.Resource set uriString = REPLACE(uriString,'http://new-ssds.mbari.org/data/ssds/ruminate/xml','http://new-ssds.mbari.org/ssdsdata/ssds/ruminate/xml') where uriString like 'http://new-ssds.mbari.org/data/ssds/ruminate/xml%'

and

UPDATE ssdsdba.Resource set uriString = REPLACE(uriString,'http://new-ssds.mbari.org/data/ssds/generated/gps','http://new-ssds.mbari.org/ssdsdata/ssds/generated/gps') where uriString like 'http://new-ssds.mbari.org/data/ssds/generated/gps%'

The second query was not necessary as it did not have any entries. After running those, I rebuilt the ssds-ruminate.jar with the updated url bases and deployed to new-ssds. Since this effectively removes all need of the http://new-ssds.mbari.org/data link, I removed that share from the http server on new-ssds as well.

While I was there

While I was updating ruminate, I changed the jboss.xml that deploys with ruminate and changed the entry:

                <MaximumSize>15</MaximumSize>

                <MaximumSize>1</MaximumSize>

Which should effectively make the RuminateMDB a singleton which should alleviate our deadlock issues that we were having (at least at Ruminate step).

Attachments:

Before Cleanup Deployment.jpg (image/jpeg)

After Cleanup Deployment.jpg (image/jpeg)

Document generated by Confluence on Feb 23, 2017 15:09